Records found: 5
Radial Attention introduces a novel sparse attention mechanism that cuts training costs by 4.4× and inference time by 3.7× in video diffusion models, enabling generation of longer videos without quality loss.
OpenBMB launches MiniCPM4, ultra-efficient large language models optimized for edge devices, featuring sparse attention and rapid inference that outperforms competitors with less training data.
Fudan University researchers have developed Lorsa, a sparse attention mechanism that disentangles atomic attention units hidden in transformer superposition, enhancing interpretability of large language models.
RWKV-X introduces a hybrid model combining sparse attention and recurrent memory to enable efficient decoding of extremely long sequences with linear complexity, outperforming previous RWKV models on long-context tasks.
Researchers from Edinburgh, Cohere, and Meta demonstrate that large sparse models can outperform smaller dense models for long-context LLMs by leveraging sparse attention, offering new scaling laws and standardized methods.